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^^ I Abstract 

Consider a compact Kahler manifold endowed with a prequantum 
bundle. Following the geometric quantization scheme, the associated 
C^ . quantum spaces are the spaces of holomorphic sections of the ten- 

sor powers of the prequantum bundle. In this paper we construct an 
asymptotic representation of the prequantum bundle automorphism 
group in these quantum spaces. We estimate the characters of these 
^ , representations under some transversality assumption. The formula 

^f) ' obtained generalizes in some sense the Lefschetz fixed point formula 

■TJ" , for the automorphisms of the prequantum bundle preserving its holo- 

\l ' morphic structure. Our results will be applied in two forthcoming 

papers to the quantum representation of the mapping class group. 

^D I Consider a compact Kahler manifold M endowed with a Herniitian holo- 

morphic bundle L ^ M whose curvature is the fundamental two-form. In 
the point of view of geometric quantization, M is the classical phase space 
and the space H^{M, L) of holomorphic sections of L is the quantum space. 

K> ' The group of holomorphic automorphisms of L acts naturally on the 

C^ ■ quantum space. Furthermore, if the higher cohomology groups {H'^{M,L), 

g ^ 1) of the sheaf of holomorphic sections of L are all trivial, the holomor- 
phic Lefschetz fixed point formula expresses the characters of this represen- 
tation in terms of characteristic classes of M and L. 

With the physical interpretation in mind, it is natural to consider the 
prequantum bundle automorphisms instead of the holomorphic automor- 
phisms. These are the automorphisms of L preserving the Chern connection 
and the metric but not necessarily the holomorphic structure. Whereas the 
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group of holomorphic automorphism is finite dimensional, the group of the 
prequantum bundle automorphisms is infinite-dimensional. Its Lie algebra 
identifies with the Poisson algebra of M. Furthermore each Hamiltonian 
symplectomorphism of M lifts to a prequantum bundle automorphism, and 
if M is simply connected, each symplectomorphism isotopic to the identity 
is Hamiltonian. The goal of this paper is to define an asymptotic represen- 
tation of these automorphisms such that a suitable version of the Lefschetz 
fixed point formula holds. Here the terms asymptotic refers to the semi- 
classical limit, obtained by replacing the prequantum bundle L by its /c-th 
tensor power with large value of k. 

It is convenient to work with the metaplectic correction. Let 5 — )• M 
be a half-form bundle, that is a square root of the canonical bundle of M. 
Such a bundle exists if and only if the second Stiefel- Whitney class of M 
vanishes. We will define the notion of half-form bundle automorphisms. 
For the introduction, it is sufficient to know that any symplectomorphism 
of M isotopic to the identity lifts to a half-form bundle automorphism. 
Furthermore on any component of M, this lifts is unique up to a sign. 

Consider two automorphisms ^l and ^ of the prequantum bundle and 
the half-form bundle respectively which lifts the same symplectomorphism 
$ of M. Then we will define a class C/(<I*l, ^) which consists of sequences 

Tk : H^{M, L^ (S,6)^ H^{M, L^ (E)6), k = l,2,... 

of unitary maps whose Schwartz kernel has a precise asymptotic. Without 
going into the complete definition, let us describe the main characteristics. 

• The Schwartz kernel concentrate on the graph of $~^ in the sense that 

Tk{y,x) = 0{k-''), ViV 
for any y and x G M such that y 7^ $(x). 

• The asymptotic on the graph is given in terms of ^l and ^ by 



The precise definition is given in sections |4.2| and |5.2| . The main properties 
of these operators are the following. 

• ?7($L,^) is not empty. For any sequences (T^) and (T^) in [/(<!> i^, ^) 
we have that 

UT^' = id+0{k~^) 

in uniform norm. 



We have an asymptotic representation in the sense that 

When the graph of ^ intersects transversally the diagonal, we can 
estimate the trace of any (T^) in U{^l, ^): 

^(^'^)= E iH.rn t\^ii/2 + 0{k-') (1) 

,i;^,)|det(id-T.c^)|i/2 

where for any fixed point x of <l>, n^; € C is the trace of the endomor- 
phism $l(3;) and lUx € Z/4Z depends only on Tx^ and ^(x). 



Precise statements are given in theorems ^.2.1 and 5.3.I . These results 



are consequences of more general statements where the half- form line bundle 



is replaced by any auxiliary line bundle, cf. theorem |4.2.2| and [4.3. 1| . How- 
ever the quantization with metaplectic correction presented here has the 
following particular features. First, we have an asymptotic representation 
of a finite cover of the prequantum bundle automorphism group, whereas 
with a general auxiliary line bundle we have to consider a C/(l)-extension. 
Second, the formula giving the asymptotic of the trace is much more compli- 
cated for a general auxiliary line bundle. It involves the complex structure 
of M in a essential way, whereas it depends only on the symplectic data in 
the half-form case. 

The index rrix appearing in the estimate of the trace (|l|) is one of the 
main point of this paper. It is similar to a Maslov or a Conley-Zehnder 
index. Our definition requires a choice of a complex polarization, whereas 
the already known definitions of indices for symplectomorphisms involved 



a choice of a real polarization. Besides the definition given in section 2.2 



we propose a simple useful characterization in section |2.3| . We also compute 
the index of the elements of the metalinear and unitary groups. 

The results of the paper relies on the articles Q and [g]. In Q we 
proposed an elementary definition of a Fourier integral operator in the con- 
text of geometric quantization of Kahler manifolds. Previously, a definition 
was given by Zelditch jl^ using the general theory of Toeplitz operators 
of Boutet de Monvel and Guillemin [g]. The interest of half- form bundle 
for these Fourier integral operators was understood in [^ where the spaces 
U{^L, ^) were introduced. The estimate of the trace with the definition of 
the index is new. Of course, it is very similar to the known formula for the 
usual Fourier integral operators. It was one of our goal to obtain the closest 
formula to the usual case. 



In a sequel of this paper we will apply our result to the quantum repre- 
sentations of the mapping class group defined in topological quantum field 
theory. Estimating the character of these representations with our formula, 
we will obtain the leading order behavior of the quantum invariant of some 
3-dimensional manifolds in the large level limit. These asymptotics were 
initially obtained by Witten in [^ by doing the perturbative theory of some 
Feynman path integral and have been rigorously proved only in few cases. 
The article [^ will be devoted to Sl(2, Z) and |Q to the mapping class group 
of surfaces with genus greater than 2. 

The layout of this paper is as follows. Sections |l], Q and ^ are devoted 
to linear algebra preliminaries. In section Q, we define the half-form mor- 
phisms and the symplectic linear category with half-forms. The automor- 
phism group of an object in this category is a concrete realization or the 
metaplectic group. It is the subject of section |2|, where the index of some 
of its element is defined. In section |3|, we generalize the previous considera- 
tions, which is necessary for the application to the quantum representation 
of the mapping class group. In section |^, we consider the quantization of 
Kahler manifold with an auxiliary line bundle, introduce the operator quan- 
tizing the symplectomorphisms and estimate their trace. In section 0, we 
treat the particular case where the auxiliary bundle is a half-form bundle. In 
appendix ^, we introduce a functor from the category of symplectic vector 
spaces with polarization and half-form to the category of Hilbert spaces. Ap- 
plying this functor to the automorphism group of a single object, we recover 
the well-known metaplectic representation. With this elementary construc- 
tion in mind, one can view the quantization of symplectomorphisms studied 
in this article as a generalization of the metaplectic representation. 

1 Half-forms 

1.1 Complex structures and canonical lines 

Let S be a symplectic real vector space. A positive polarization ii^ of S* is a 
Lagrangian subspace of 5 (8) C such that 

jU}{x,x) > 

for any non-vanishing x € E. Any positive polarization has a canonical 
Hermitian scalar product given by 

{x,y) -^ jU}{x,y). 



The set of positive polarizations of 5" is a contractible topological space. 

For any positive polarizations, we consider the canonical line a*°p£'* 
with its associated Hermitian product. For any positive polarizations Ea 
and Eh, S C is the direct sum of Ea and Eb. Let '^Ef,,Ea ■ Eb ^ Ea be the 
restriction of the projection onto Ea with kernel Eb- This map is invertible. 
Define 

'^E.,E,=7r%^,E.:A'^^E:^A'°^E; 

Given three positive polarizations, let Ci^^a, Eb,Ec) be the complex number 
such that 

^E^,E^ = C{Ea, Eb, Ec)^E„E^ o ^ e^,E, (2) 

Define C^ (^aj Eb, Ec) to be the square root depending continuously on Ea, 
Eb and Ec and taking the value 1 when Ea = Eb = Ec- 

1.2 Half- form lines 

For any positive polarization E, a half- form line of i? is a complex line 5 
together with an isomorphism if : 5^'^ — ?> A^°^E*. We endow S with the 
scalar product making ip a unitary map. 

Let us consider the category T>{S) with objects the triples {E, 6, ip) con- 
sisting of a positive polarization together with a half-form line. The mor- 
phisms from {Ea,5a,^a) to {Eb,5b,(pb) are the linear maps "^ : 6a ^- St 
satisfying 

ipbO^iJ^'^ = ^Ea,E,Oipa. 

The composition of a morphism ^ : {Ea,6a,'-Pa) -^ iEb,Sb,(pb) with a mor- 
phism ^' : (Eb, 6b, (fb) — ^ {Ec, 6c, (fc) is defined as 

^i>'ojy^:=Cy\Ea,Eb,Ec)^'o^ 

where the product in the right-hand side is the usual composition of maps. 

Proposition 1.2.1. The product o-p is well-defined and associative. For any 
object {E,j, if), the identity of 6 is a unit of {E, 6, ip). So P(S') is a category. 
Furthermore each morphism is invertible, its inverse being its adjoint. 

Proof. Because of equation (|2|), ^' o-p ^1/ is a morphism. We deduce from the 
associativity 

*£,,£, o {^E„E, o ^E,,E,) = {^ E,,E, o ^ E„eJ o ^ E,,E, 



the cocycle identity 

C{Eb, Ec, Ed)C{Ea, Eb, Ed) = C{Ea, Ec, Ed)C{Ea, Eb, Ec). 

The square root satisfying the same identity, the product o-p is associative. 
Since ^e,e is the identity map, the identity of (5 is a half-form morphism. 
Furthermore, 

'^Et,Et ° '^Ea,Et = '^Ea,Et, '^Ea,Et ° '^Ea,Ea = "^ Ea^E^ 

imply that 

C,{Ea, Eb, Eh) = C{Ea-, Ea, Ef,) = 1. 

So the square root of C satisfies the same equation, consequently the identity 
of 5 is a unit of {E, 5, if). 

Since Ea and Ef, are Lagrangian spaces, one has 

^{x-'n-Ea,EtX,7rE^,Eay) =0, (^{TTEa,Et,X,y-7rEt,Eay) = 

for any x £ Ea and y G Ei,. Consequently 

Uj{x,TTE,,Eay) = (^iT^Ea,E,X,y) 

which proves that T^Ei„Ea is the adjoint of 7rEa,Ei,- So '^Et,Ea is the adjoint 
of ^Ea,E,,- Hence the adjoint ^b,a of a half-form morphism ^a,b is a half- 
form morphism. Furthermore, since the only automorphisms of a half-form 
bundle are id and — id, one has 

^6,a o© ^a,fe = ± id, ^a,bOB^6,a = ±id. 

Deforming from Ea = E^ to remove the sign ambiguity, we obtain that ^b,a 
is the inverse of ^a,6- D 

1.3 Symplectic linear category with half- forms 

We introduce now a category V with objects the quadruples {S, E, 5, ip) 
consisting of a symplectic vector space with a positive polarization and a 
half-form line. Let us define the morphisms. 

Consider two symplectic vector spaces Sa and Sb with positive polariza- 
tions Ea and Eb respectively. A symplectic linear isomorphism g from Sa to 
Sb sends isomorphically Ea to the positive polarization gEa of Sb- So com- 
posing the pull-back by g~^ with the morphism '^gEa,Et from A^°^{gEa)* to 
A*°P£'^, we obtain an isomorphism 

^,,E.,E, := ^,E.,E, o {g~')* : a'^^EI ^ a'^^E^. 



The morphisms from a = {Sa,Ea,5a,^a) to b = (Si,, Eh, 6b, (fb) are defined 
as the pairs {g, ^) consisting of a symplectic linear isomorphism from Sa to 
Sb and a linear morphism 5a -^ Sb such that 

The composition of a morphism (g,^) : a ^ b with a morphism {g',^') : 
6 — > c is defined as 

where the product in the right-hand side is the usual composition of maps. 

Proposition 1.3.1. T> is a category where each morphism is invertible. 

Proof. This follows from proposition |1.2."l] . Indeed, given a morphism ((/, ^) : 
a ^>- b, one may identify Sa and Sb with S through symplectomorphisms 
in such a way that g becomes the identity. Then ^ is a morphism from 
{Ea,6a,^a) to {Eb,5b,(pb) in the category V{S). With these identifica- 
tions, the composition of morphisms in T) corresponds to the composition 
\nV{S). D 

In appendix ^, we will define a functor from the category T> to the 
category of Hilbert spaces. 

2 Metaplectic group 

2.1 The automorphism group 

Consider a fixed symplectic vector space S together with a positive polar- 
ization E. Denote by Sp(«S') the group of linear symplectomorphism. For 
any g G Sp(«S'), observe that the endomorphism ^g^E,E of A*°P-E* is the 
multiplication by <iei{g~^TTE,gE : E ^ E). 

Let ((5, (/?) be a half-form line of {S,E). Then identifying the automor- 
phisms of 6 with complex numbers, the automorphisms of {S, E, 6, ip) are 
the pairs {g, z) consisting of a linear symplectomorphism g G Sp(S') with a 
complex number z such that 

z^ = det{g-^TrE,gE : E ^ E). 

The composition of two automorphisms is given by 

{g',z')oj,{g,z) = {g'gX^'\g'gE,g'E,E)z'z). 



Since the previous formulas don't depend on {6,ip), we denote by Mp{S,E) 
the group of automorphisms of {S, E, 6, 99). The projection 

Mp{S,E) ^Sp(5) 

is two to one. Recah that Sp(S') is connected with fundamental group Z. 
The metaplectic group is defined as the 2-cover of the symplectic group. 

Proposition 2.1.1. The group Mp{S,E) is connected, so it is isomorphic 
to the metaplectic group. 

Proof. Choose a line D of E. Then for any 6 S R, consider the symplecto- 
morphism gg which preserves D and E and such that for any x d E, 

{exp(z0)x if X G D 

■f I n 
X 11 X ± D 

Then 9 — )■ {gg,exp(—i0/2)) is a path of automorphisms connecting (id, 1) 
with (id, -1). D 

2.2 Definition of the index 

Consider the complex structure j of S such that ker(j — iid) = E. Then 
{X,Y) -^ cl!{X,jY) is a scalar product of S. Let Sym(S', j) be the space of 
linear endomorphisms of S symmetric with respect to this scalar product. 
For any A € Syioa{S,j), define the square root 

det^/^{^id+iA) 

in such a way that it depends continuously on A and is positive for A = 
0. It is well-defined because Sym(5, j) is contractible, as a vector space. 
Observe also that any symmetric endomorphism A is diagonalisable with a 
real spectrum so that 2 id +iA is invertible. 

Let Sp^(5') be the set of symplectic linear isomorphism g of S such that 
id —g is invertible. For any such g, it is easily checked that 

A{g) := ^(id+g) o (id -gy^ o j 

belongs to Sym(5, j). Denote by Mp^(S', E) the subset of Mp(S', E) consist- 
ing of the pairs {g, z) such that g G Sp^(S'). 

Proposition 2.2.1. For any automorphism {g,z) S Mp^{S,E), one has 

jm{g,z) 

where m{g,z) G Z/4Z. 



Proof. Since the inverse of (id—g) ^j is —j{id—g), one has 

i id +iA{g) =i (-i(id -g) + i(id +5)) (id -<7)"^J 
= -j(vf-7r5f)(id-5)~^j 

where vr = 2(id— ij) is the projector of 5 (8) C with image E and kernel E. 
From this one deduces that 

det(iid+z^(,)) = ^^^^^r^ = (-ir ^^^i"^;f^^^ 

^2 ^^^^ det(id-c/) ^ ' det(id-5) 

The inverse of t^e.qE is the restriction of vr from gE to E. So 

z2 = det(5~V£:,g£: : E -^ E) = det~^(7r5 : E -^ E). (3) 

The two previous equations imply that 

(-1)"' 
.^det(lid+ato))=;jAj-jL_. (4) 

which concludes the proof. D 

We call m(g, z) the index of {g, z). This defines a locally constant map m 
from Mp^(S', E) to Z/4Z which distinguishes the components of Mp^(5, £"), 
as proves the following proposition. 

Proposition 2.2.2. Mp^(S', iiJ) has four components: r7T,~^(0), m~^{l), 
m~^(2) and m~^(3). 

Observe also that m(— id, e*'2^) = p for p = n mod 2. When 5 is two- 
dimensional one can can easily characterize the index of (^f, z) in term of the 
trace of g and the argument of z, cf. equation (^). 

Proof. It is proved in |8| lemma 1.7, that Sp^(5) has two components, distin- 
guished by the sign of det(g' — id). So Mp^(5, E) has at most 4 components. 
To conclude it suffices to prove that m is onto. This follows from equation 
(§), because the right hand side can be positive and negative. Actually, it 
is also proved in H] lemma 1.7 that any loop in Sp^(S') is contractible in 
Sp(5), which gives another proof that Mp^(5, ii^) has 4 components. D 
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Figure 1: the index 



2.3 A characterization of the index 

Assume S is two-dimensional. By choosing a basis (e, /) of S such that the 
symplectic product of e with / is equal to 1, we identify S with M^ and 
Sp(5) with S1(2,M). For any g G Sp(5), 

det(id-5) = 2-tr((7). 

So the set of g with det(id —g) < consists of the hyperbolic elements with 
negative trace, whereas the set of g with det(id —g) > consists of the 
elliptic elements together with the hyperbolic ones with a positive trace. 

Recall that S1(2,]R) is diffeomorphic to the product of the circle and the 
unit disc. Such a diffeomorphism is the map sending 6 £ S^ and (u, v) €z D 
into 



g{9,u,v) := (l 



{{u,v)/ u^ + v^ < 1}. 



-1/2 fcos{9) + u 
sm{6) + V 



— sm{9) + V 
cos{9) — u 



where D 

Let E be the positive polarization generated hy e — if and let Mp(5, E) 
be the associated metaplectic group. Let us parametrize Mp(5, E) hy S^ xD 



i9,u,v)^{gi-29,u,v),e''{l 



ThenMp^(S',£') is the image of {cos (20) / {l-u^-vY }. We find easily 
the index of any element of Sp^(5) by computing explicitly the index of one 
element in each component, cf. figure ||. 

Observe that when the argument of z runs over an interval [/c^, (A;-|-l)^ [ 
with /c € Z, the index m{g, z) takes two distinct values depending on the sign 
of tr(5) — 2. Checking the various cases, one obtains the following formula 



m{g,z)=k + Ul-{-l)^+^) 



(5) 



10 



where k and e are determined by 

arg(z)G A;f,(A; + l)f , e= ^^[ 

I 1 otherwise. 

The formula does not depend on the parametrization. Unfortunately 
such a simple description doesn't generalize in higher dimension. Neverthe- 
less we can characterize the index in any dimension by considering product. 

Let Si and 52 be two symplectic vector spaces with positive polarizations 
El and £'2- Then E = Ei x E2 is a positive polarization of S = Si x 5*2. 
We have a morphism from Mp(5i,-Ei) x Mp(S'2,-E2) to M-p{S,E) sending 
{{gi,zi), {92, Z2)) into (51 x g2,ziZ2). Furthermore, 

migi X 52, Z1Z2) = m{gi,zi) + m{g2, Z2). (6) 

So the image of Mp^(S'i,E'i) xMp^(S'2, E2) meets each connected component 
of Mp^(S', £^). This gives the following characterization. 

Proposition 2.3.1. The collection {ms^E '■ ^v{S,E) — )■ Z/4Z), where 
{S, E) runs over the symplectic vector space endowed with a positive po- 
larization, is the unique collection of continuous map satisfying ^ for any 
two dimensional space and ^ for any product. 

2.4 Unitary group of E 

The subgroup of Sp(5) consisting of the elements commuting with j is iso- 
morphic with the unitary group of E. The isomorphism is the map l sending 
h G U{E) to g G Sp(5) whose complexification acts as 

, , i h(x) if X £ E 

q(x) = < _ 

\h{x) ifx£E 

Denote by U^{E) the subset of U{E) consisting of the h such that h — id is 
invertible. Obviously h G U^{E) iff L{h) G Sp^(5). Next lemma will be used 
to compare our trace estimates with the holomorphic Lefschetz fixed point 
formula. 

Lemma 2.4.1. For any h G U^{E), we have 

de.V^(iid+aW.)))=5jJi^. 
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Proof. Let (cj) be an orthonormal basis of E diagonalising h. Then the 
matrix of A{g) in the base (ei, . . . , e^, ei, . . . , e„) is given by 

D 
D 

with D the diagonal matrix with entries di = |(l + Ui)(l — Uj)"^. Here the 
Uj's are the eigenvalues of h, h{ei) = UiCi. A straightforward computation 
using that Uj is a complex number with modulus 1 gives 

(i + i(ij)(i + id,' 



so that 

This proves the result up to a plus or minus sign. U^{E) being connected, it 
suffices now to check the result for one element h. It is obvious for h = —\d, 
because A{i{h)) = 0. D 

Let U{E) be the subgroup of U{E)xC 

U{E) = {{h,z)/ z^ = deth} 

It is isomorphic to the twofold cover of U{E). By equation (H), l lifts to the 
embedding from U{E) to Mp(S', S) sending {h,z) to (t(/i),z~^). This map 
is a group morphism. Finally observe that U^:{E) has 2 components, one 
containing (— id, e) and the other (—id, — e) with e^ = (—1)". So the index 
m takes two distinct values on U.^{E). 

2.5 Metalinear group of a Lagrangian subspace 

Let A be a Lagrangian subspace of S. Let Sp(S', A, jA) be the subgroup of 
Sp(S') consisting of the elements preserving A and jK. Let M1(A) be the 
metalinear group of A, i.e. the subgroup of G1(A) x C* consisting of the 
pairs (/i, z) such that z^ = det h. 

Proposition 2.5.1. For any (h, z) € M1(A), there is a unique pair {g,z') € 
Sp(S', A,jA) X C* such that 

• h is the restriction of g to A. 

• z'/z G M+. 
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The map j'a : M1(A) — )> Mp(S', ii^) sending {h,z) into {g,z') is an injective 
group morphisms, with image the set of {g,z') such that g € Sp{S,A,jA). 

Proof. Let (ej) be a basis of A, orthonormal for the scalar product u}{x,jy). 
Then g G Sp(«S', A,jA) if and only if its matrix in the basis ei, . . . ,e„, jei, 
. . ., jCn is of the form 

^ '] (7) 

c) ^ ' 

with BC^ = id. So g is determined by its restriction h to A which is an 
arbitrary element of G1(A). 

Denote by n the projection onto E with kernel E. Then the matrix of 
irg : E ^ E in the basis (e^ — ijei)i is ^{B + C). Furthermore det(i? + C) 
and det-B have the same sign. So det^irg : E —^ E) is real and have the 
same sign of det(/i). This proves the existence and unicity of z' . 

To prove that j'a is a morphism, we have to show that (^'"^{g'gE, g'E, E) 
is a positive number for any g,g' G Sp(5, A,jA). By the first part of the 
proof, C{g'gE, g' E, E) is real. So it suffices to prove it is positive for one pair 
(/i, h') in each component of G1(A) x G1(A). If any two of three polarizations 
E^ F and G are equal, then 

e'\G,F,E) = l. 

One deduces that C^/^{g'gE,g'E,E) = 1 for {h,h') = (id, id), (id, A:), (A:, id) 
and {k, k) where k is any involution of A with negative determinant. D 

Proposition 2.5.2. For any {h,z) G M1(A) such that h — id is invertihle, 
the index m of j\{h,z) is determined by z = i™|z|. 

Proof. One has to prove that det ' (^^ id -\-iA{g)^ is positive for any g G 
Sp(5, A,jA). If the matrix of g is (^), the one of A{g) is 



i/O D 
.D* 



-2, , ^ , withD = {l + B){l-By 



So 



det {I id +iA{g)) = det(i(id+D*L>)). 



Deforming A{g) to through a radial homothety we obtain that the square 
root of det(^ id+ij4((7)) is positive. D 
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3 Generalized half-form lines 

Let p be a positive integer. A generalized half-form line of a symplectic 
vector space S equipped with a positive polarization £^ is a complex line 6 
together with an isomorphism 

We have a category whose objects are the quadruples {S, E, 6, ip). The mor- 
phisms from {Sa, Ea, 6a, ^a) to [Sh, Ei,, 6b, ipb) are the pairs consisting of a lin- 
ear symplectomorphism g : Sa ^ Sb together with a morphism "$ : 6a ^ 6b 
such that 

(^ft u w ^g,Ea,Et No- 

where the map ^g^Ea,Et is defined as in section |l.3| . The composition of a 
morphism (g, ^) : a — ?• 6 with a morphism ((7', ^') : 5 — )• c is defined as 

{9',^') ov (5,^) := {g'g,C}^\g'gEa,g'Eb,E,)^'o^>) 

where the product in the right-hand side is the usual composition of maps. 
The automorphism group Mpp(5, -E) of {S,E,6,ip) consists of the pair 
{g, z) where 5 is a linear symplectomorphism of S and z a complex number 
such that 

z^P = detP{g-^TrE,gE--E^E). 

Let U2p be the group of 2p-th roots of unity. The map sending {{g, z),u) to 
{g, zu) is a surjective morphism 

Mp{S,E)xU2p^Mpp{S,E) 

with kernel {((id, 1), 1), ((id, -1), -1)}. For any element {g,z) G Mpp(S',£') 
such that 1 is not an eigenvalue of ^f, we define its index in the following 
way. Set p' = p il p is even and p' = 2p \i p is odd. Then mp{g, z) is the 
unique element of Z mod 2p'Z such that 



:det^/^{^ id +iA{g)) 



^ifrmp{g,z) 



det(id-g)|i/2 
The existence of mp{g, z) follows from proposition p. 2.1 
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4 Quantization of Kahler manifolds 

4.1 Hilbert space 

Consider a compact Kahler manifold M with a prequantization bundle L, 
that is L is a holomorphic Hermitian line bundle such that the curvature of 
its Chern connection is ^u where w is the fundamental two- form of M. Let 
X — > M be a holomorphic Hermitian line bundle. Define the sequence of 
vector spaces 

l-Lk '•= {holomorphic sections ofL iSi K^, k = 1,2, ... 

Since M is compact, Tik is a finite dimensional vector space. It has a natural 
scalar product defined by means of the Hermitian structure oi L^ ® K and 
the Liouville measure of M. 

4.2 Fourier integral operators 

Consider a symplectomorphism ^ : M ^^ M together with an automorphism 
$i of the bundle L lifting $ and preserving the connection and the metric. 
To these data is associated a space J-{^l) of Fourier integral operators, that 
we define now. 

Consider a family of operators {S^ ■ Tik —^ T~(-k^ k = 1,2, . . .). The scalar 
product of 1-ik gives us an isomorphism 

Hom(-Hfc,-Hfc)~-Hfe«)Wfc. 

The latter space can be regarded as the space of holomorphic sections of 

where M^ is endowed with the complex structure (j, — j). The section 
Sk{x,y) associated in this way to Sk is its Schwartz kernel. 
By definition {Sk) is a Fourier integral operator of J-{^l) if 

Skix, y) = (— ) F'ix, y)g{x, y, k) + 0{k-^) (8) 

where 

• F is a section oi LML ^ M^ such that ||F(x,2/)|| < 1 if x / ^{y), 

F{(t{x),x) = ^l{u) (8)u, Vn e Lx such that ||n|| = 1, 

and BF = modulo a section vanishing to any order along the graph 
of -l--^ 
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• g{.,k) is a sequence of sections of K M K -^ M^ which admits an 
asymptotic expansion in the C°° topology of the form 

9{; k) = go + k~^gi + k~'^g2 + ... 

whose coefficients satisfy dgi = modulo a section vanishing to any 
order along the graph of <I>~^. 

Let us define the principal symbol of (Sk) to be the map x -^ gQ{^{x), x). 
Using the Hermitian structure of K, we regard it as a section of the bundle 
Hom(i^, 'I'^-ftr ) -^ M. The principal symbol map 

a : 7-($l) ^ C°°{M, Rom{K, $,K)) 

satisfies the expected property. 

Theorem 4.2.1. The following sequence is exact 

-^ T{^l) n 0{k~^) -^ T{<^l) ^ C°°(M,Hom(A", ^^A")) -^ 0, 

where the 0{k^^) is for the uniform norm of operators. 

Consider two symplectomorphisms $ and $'. Define the product of 
two sections ^, ^' of Hom(i^, $*Ar) and Hom(Ar, <l>'^Ar) respectively as the 
section of Hom(i^, (<!>' o ^)^,K) given at x by 

^' o^ ^>{x) = C^/2((7$(,)5x^x,5$(.)^${x),^(4.'o<i.){x))*'($(2;)) o ^(x) (9) 

where Ey = Ty'^M, gy = Ty^ and g'y = Ty^' for y = x, $(x) or ^'(^(x)). 
Here the product on the right hand side is the usual composition of homo- 
morphism. 

Theorem 4.2.2. Let ^l and <1>^ be two automorphisms of the prequantum 
bundle L lifting $ and <!>' respectively. IfTG J^{^l) and S G T{^'^), then 
S oT is a Fourier integral operator of J^{^'^ o ^i). Its symbol is given by 

a{SoT) = a{S)o^a{T). 



Theorems 4.2.1 and 4.2.2 are consequences of theorems 3.1 and 3.2 of 
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4.3 Trace estimate 

Consider a symplectomorphism $ of M together with a hft ^l to the pre- 
quantum bundle. Assume that the graph of <l> intersects transversally the 
diagonal. 



Theorem 4.3.1. For any (Sk) € J-{^l) with symbol ^ , we have 
T^{Sk)= Yl ^xdeti/2(iid+i^,)n^ 

x=$(x) 

where for any fixed point x of M, 

• Zx and Ux are the traces of ^(x) : K^ -^ K^ and $l(x) : L^ — )> L^ 
respectively. 

• Ax = i(id+r^<l>) o (id — T^^)"-*^ o j^ and the square root of the deter- 



minant is determined as in section 2.^. 
Proof. By assumption the Schwartz kernel of S has the form (^). One has 

Tr(5fc) = (A)"y F''ix,x)fix,x,k)fiMix) + Oik-^) 

Since \F{x,y)\ < 1 outside F = {($(x),x)/ x G M}, one can restrict the 
integral to a neighborhood of the fixed point set of $. Let us write on a 
neighborhood of F 

In the proposition 2.2 of ^, the first derivatives of /? along F are computed. 
Denote by Ex the space Tx' M. 

Lemma 4.3.2. The form (3 vanishes along F. For any vector fields X and 
Y of M"^ , one has at any point ofV 

where ujmxM- is the symplectic form of the product of (M, w) with (M, —uj). 
And for any x G M , q(^^(^x),x) is the projection ofT^(^x)MxTxM onto E^^^x) ^ 
Ex with kernel the tangent space ofV at ($(a:;),x). 

Let us write on a neighborhood of a fixed point xq of $ 

F{x.,x) = exp(— y3(x)) 



where (p \s a complex valued function. By lemma [4.3.2 , the first derivatives 
of (/? vanishes at xq. 
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Lemma 4.3.3. The Hessian of ip at xq is given by 

Ressip{X,X'){xo) = uj{{^id+iA^,)-'x,jX') 

for any tangent vectors X,X' of M at xq. 

Proof. By lemma |4.3.2| , the Hessian is given by 

Hess ^{X, X'){xo) = iu^MxM- {q{X, X), {X' , X')) 

Let IT be the projector of T^M ® C with image Ex and kernel Ex- Then 

qiX,X) = iw{V),7riV)) 

for a unique V € T^M (E) C Using that vr = 2 (id —ijx), we obtain that 

Hess ip{X, X'){xo) =iuj{7r{V),X') - iio{7r{V),X') 
=u{V,jxX') 

To compute V one has to solve the following system 

(x = TxHY) + l{[d+ijx)V 
\x = Y + ^{id-ijx)V 

Adding and subtracting both equations, we obtain 

(2X = {id+Tx<^){Y) + V 
\o = {id-Tx<^)iY)-iJxV 

With the second equation, we compute Y in terms of V. Liserting the result 
in the first equation, we obtain 

V={lid+iAxo)~^X 

with proves the lemma. D 

Now the theorem follows from stationary phase lemma by using that 
exp(— (/?(xo)) = UxQ and g{xo,XQ,k) = Zx^ + 0{k~^). Observe further- 
more that the Liouville measure is the Riemannian volume of the metric 
uj{X,jxY). So for an orthogonal basis Xi, . . . , X2„, we have 

^J,M{Xl^...^X2n) = i 

and 

det(Hess(^(Xi,Xj)(xo)).^^. = d&i~^[\ id +iAx^) 

which leads to the result. D 
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4.4 The holomorphic Lefschetz fixed point formula 

As in the previous section, consider a symplectomorphism $ of M together 
with a hft ^L to the prequantum bundle. Assume that $ is a holomorphic 
map. Since the holomorphic structure of L is characterized by the connec- 
tion, ^L is a holomorphic bundle homomorphism. Consider a holomorphic 
bundle homomorphism ^ of i^ lifting $. Then we have a map 

defined as the inverse of the pull-back by ^\ (8) ^ ■ More generally, $^ (g) ^ 
acts on the q-ih. cohomology group of the sheaf of holomorphic sections of 
L^ <^ K. One defines the holomorphic Lefschetz number 

n 

Then assuming that the graph of <1> intersects transversally the diagonal, the 
holomorphic Lefschetz fixed point theorem, cf. |j^] theorem 4.12, says 

,i;^,)det(id-/i. ) 



where the complex numbers z^ and u^ are defined as in theorem 4.3.1 and h. 



is the holomorphic tangent map of $ at j;, that is the restriction oiTx^ ®C 
to E^ = Tt^M. 

When k is sufficiently large, Kodaira's vanishing theorem implies that 
H'i{M,L^ (8) K) = for every positive q. If the latter is the case, the 
holomorphic Lefschetz number is the trace of the action of <1>2, (8) ^ on 71^. 

Furthermore, the family of operators {{^'l ® ^)*|-^^, ,/c = 1,2, . . .) is a 
Fourier integral operator of J^($i) with symbol ^. This is an easy conse- 
quence of the fact that the sequence (id-^^., A: = 1, 2, . . .) belongs to J^(idL) 
and has the symbol id;^. 

So theorem [4.3.1 gives the asymptotic behaviour of the Lefschetz num- 



bers. That the result agrees with holomorphic Lefschetz fixed point theorem 



is a consequence of lemma 2.4.1 



5 Quantization with half-form bundle 

5.1 Hilbert spaces 

Let p be a positive integer. Let {6, (p) be a generalized half-form bundle of 
M, i.e. (5 is a complex line bundle over M and (p is an isomorphism form 6®"^^ 
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to (A"''^r*M) . So at each point x € M, we have a positive polarization 

ri'OM = ker(j, -iid) 

of TxM and a generahzed half- form line {6x,^x) of this polarization. 

The half-form bundle 6 has a natural metric and holomorphic structure 
such that if is an isomorphism of Hermitian holomorphic bundle. We apply 
the previous constructions with K = 6, which defines the Hilbert space Tik- 

5.2 Unitary maps 

Let ^ be an automorphism of the bundle 6 lifting a symplectomorphism $ 
of M. One says that ^ is a half- form bundle automorphism if for any point 
X of M, 

is a morphism of half- form lines, cf. section L3 for p = 1 and section ^ for 



any p. Observe that for any two half-form bundle automorphisms ^ and ^', 
the product '^' o-p ^ defined in (P) is a half-form bundle morphism. 

Theorem 5.2.1. For any automorphisms <&l,^ of the prequantum bundle 
L and the half-form bundle 6 respectively which lift the same symplectomor- 
phism of M, let U{^L,^) be the set of unitary Fourier integral operators of 
J-{^l) with symbol ^. Then 

• [/(<^L,^) is not empty. 

• n e U{^L,^) and Tl e C/(^'^, ^') ^ {T'^Tk) G u{^'^ o $i, m' o^ m) 

• [/(idijid^) consists of the sequences ex.p{ik~^Tf:) where (T^) runs over 
the self-adjoint operators of J^{[dL). 

Proof. We only give an outline since the ideas of the proof are standard. To 
show that U{^L,^) is not empty, consider a Fourier integral operator (T^) 
of J-'{^l) with symbol ^. Then its adjoint is a Fourier integral operator of 
J-'($^ ) with symbol ^*. By proposition |1.2.l] , ^* is the inverse of ^. So by 
theorem 4.2.2, {T^TjS) is a Fourier integral operator of J-'(id/,) with symbol 



the identity. J^(idi) is the algebra of Toeplitz operators. By ellipticity, 
T^Tk is an invertible self-adjoint operator when k is sufficiently large. By 
changing the first values of T^, T^T}^ is invertible for any k. Then using 
the functional calculus of Toeplitz operators (cf. proposition 12 of [^), one 
proves that (T^Tfc)"^'^ is a Toeplitz operator with principal symbol equal 
to 1. This implies that [Tk{TlTk)-^/'^) belongs to C/($l,^). 
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The second part of the theorem follows directly from theorem 4.2.2. To 
show the last part, one constructs the operator T^ by successive approxima- 
tions using the functional calculus of Toeplitz operators. D 

5.3 Trace estimates 

Consider two automorphisms ^l,^ of the prequantum bundle L and the 
half-form bundle 6 respectively which lift the same symplectomorphism $ 
of M. 

Theorem 5.3.1. Assume that the graph of <& intersects transversally the 
diagonal of M'^. Then for any (T^) € C/($l,^), one has 

where for any fixed point x of ^, 

• gx is the linear tangent map to ^ at x and Zx (z C is the trace of the 
endomorphism ^s,x '■ ^x ^f' ^x 

• p' = p (resp. 2p) if p is even (resp. odd) and mp{gx,Zx) € Z mod Ip'TL 
is the index defined in section |^. 

• Ux (z C is the trace of the endomorphism ^l,x '■ Lx ^ Lx 



This is an immediate consequence of theorem [4.3. 1| and the definition of 
the index. 

A Linear Quantization 

In this appendix we define a functor from the category of symplectic space 
with polarization and half-form to the category of Hilbert space. 

A.l Hilbert space 

Let S* be a symplectic vector space. Consider the trivial bundle Lg with 
base S, fiber C and endowed with the connection d -\- ja where a G Q^{S) 
is given by 

a\x{y) = |w(x,y). 
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Let -E be a positive polarization and {5, (p) be a half- form line. Abusing 
notation, we denote by 5 the trivial bundle with base S, fiber 5 and endowed 
with the trivial connection. 

Consider the space ^{{S^E^S^^p) which consists of the holomorphic sec- 
tions ^ of Ls ® S with respect to the polarization E such that 

*(a::)p/i(x) < oo 

s 

where ^ the Liouville measure of S. Here |^l'(x)| denote the punctual norm 
in Ls <S) S ■ That a section ^ is holomorphic with respect to E means that 
its covariant derivative with respect to any vector of E vanishes. 

His, E, 6, if) is an abstract presentation of the Bargmann space. It is a 
Hilbert space with the scalar product j'^(^,^')(x) /i(x). 

A. 2 Unitary map 

Consider symplectic vector spaces (5a, Ea) and [Sh-, E^) with positive polar- 
izations. Let g be a linear symplectomorphism from Sa to St,. 

Lemma A. 2.1. There exists a unique quadratic function ^ : Sh x Sa —?■ C 
vanishing on the graph of g and such that exp($) is a holomorphic section 
of Ls^ Kl Ls^ with respect to Ei, x Ea ■ 

Consider now a half- form line {6i,fi) of Ei for i = a,b. Then for any 
morphism (g, ^) from a = {Sa, Ea, 5a, ^a) to 6 = {S^, Ef,, 5b, ipb) we define a 
map from 7^ (a) to 7^(6) by 

([/(5,^)/)(x) = (27r)-" / exp(#(x,y))^(/(y))//b(y) 

JSa 

where fi^ is the Liouville measure of S^- 

Theorem A. 2. 2. For any morphism (5,^), the operator U{g,^) is uni- 
tary. Furthermore the map sending {S,E,5,ip) to T-L{S,E,5,ip) and (5,^) 
to U{g,^) is a functor from the category V to the category of Hilbert spaces. 

The elementary but long proof of this result will be provided somewhere 
else. Applying the functor to the automorphism group of a symplectic space 
with polarization and half-form, we obtain the well-known metaplectic rep- 
resentation. 
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